NASA Case No. MFS-31 865-1 PATENT APPLICATION 

CONTROL METHOD FOR VIDEO GUIDANCE SENSOR SYSTEM 

Origin of the Invention 

[0001] This invention was made by employees of the United States 
Govemment and may be manufactured and used by or for the Government for 
governmental purposes without the payment of any royalties. 

Field of the Invention 

[0002] The present invention relates to a video guidance sensor system for 
space vehicles and, more particularly, relates to a software-based control method for 
such a system. 

Background of the Invention 

[0003] Prior techniques used in determining the range between two 
spacecraft vehicles for automatic rendezvous and docking of such, include vehicle 
radar, man in loop estimates, global positioning systems, lasers, loran, and video 
guidance sensor systems for processing optical images in determining range. The 
video guidance sensor system approach, which is of particular importance here, is 
based on the concept of using captured and processed images to determine the 
relative positions and attitudes of a video guidance sensor and target. 
[0004] One prior art video guidance sensor system uses a pair of lasers of 
predetermined wavelengths to illuminate a target. The target includes a pattern of 
filtered retroreflectors to reflect light. The filtered retroreflectors pass one 
wavelength of light and absorb the other. Two successive pictures or images are 
taken of the reflected light and the two images are then subtracted one from the 
other, thereby allowing for target spots to be easily tracked. Such a system Is 
described, for example, in R. Howard, T. Bryan, M. Book, and J. Jackson, "Active 
Sensor System for Automatic Rendezvous and Deciding'' SPIE Aerosense 
Conference, 1997, which is hereby incorporated by reference. 
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[0005] Another prior art video guidance sensor system uses a CMOS imaging 
chip and a digital signal processor (DSP) in order to provide higher-speed target 
tracking and higher-speed image processing. The high-speed tracking rates result 
in a more robust and flexible video guidance sensor. Because of these faster 
tracking rates, the video guidance sensor system can track faster moving objects or 
provide more data about slower moving objects. Such a system is described, for 
example, in R. Howard, M. Book and T. Bryan, ''Video-based sensor for tracking 3- 
dimensional targets" Atmospheric Propagation, Adaptive Systems, & Laser Radar 
Technology for Remote Sensing, SPIE Volume 4167, Europto Conference, Sept. 
2000, and in R. Howard, T. Bryan, and M. Book, "The Video Guidance Sensor: 
Space, Air, Ground and Sea" GN&C Conference, 2000, which are also hereby 
incorporated by reference. 

[0006] An improved video guidance system is described in R. Howard, T. 
Bryan and M. Book, "An Advanced Sensor for Automated Docking,*' Proceedings of 
Digital Avionics Systems Conference (October, 2001), which is also hereby 
incorporated by reference. The basic components of the overall docking system 
include an on-board computer, control software for calculating the correct thruster 
firings necessary to achieve docking, a sensor for long-range (rendezvous) 
operations, a sensor for short range (proximity) operations, and grapple mechanism 
or fixture for allowing the system to be attached to the target vehicle. The video 
guidance sensor serves as the short range sensor and must be able to handle the 
transition from rendezvous to proximity operations and to pick up where the 
rendezvous sensor leaves off. 

[0007] The improved video guidance sensor is generally based on the 
tracking of "spots" on known targets as explained below. By tracking only spots, a 
large amount of image processing/scene interpretation is avoided, and because the 
spots occupy known positions on the target, the sensor can easily determine relative 
position and attitude data based on the centroids of the spots. 
[0008] The basic video guidance sensor system consists of the sensor and 
the target. The sensor is the active part and includes a pair of laser illuminators 
which illuminate the target and an image processor which processes the return 
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images. In one preferred embodiment, the target is an arrangement of corner-cube 
retro-reflectors with associated optical filters. 

[0009] The improved video guidance sensor currently uses lasers of two 
different wavelengths to illuminate the target. One of the wavelengths passes 
through the filters in front of the retro-reflectors on the target while the other 
wavelength is absorbed by the filters. The system essentially functions by taking a 
picture of the target illuminated by the foreground lasers and then taking a picture of 
the target illuminated by the background lasers. The second picture is subtracted 
from the first, thereby producing a very low-noise image with some bright spots at 
the target reflector locations. After the image subtraction, the processor (DSP) 
processes the resulting intensity data, essentially by assembling the pixels in the 
image which are above a preset threshold into "blobs." The blobs are then screened 
for size, the centroids of the blobs are computed and the relation between the 
positions of the blobs is used to determine whether there is a match between the 
blob pattern and the known target pattern. 

[0010] Once the target "spots" are picked out of all of the blobs, tracking 
windows are established around the target spots and the relative positions and 
attitudes between the sensor and the target are computed. When the target is close 
enough to the sensor, tracking windows are established around each of the target 
spots. The acquisition cycle or mode requires processing the entire image to find 
the target. The use of tracking windows has the advantage that once the target has 
been acquired, only the area around the target spots must be processed each cycle, 
so that processing time is decreased. 

Summarv of the Invention 

[001 1 ] According to the invention, a software-based control method is 
provided for controlling implementation of, and transitioning between, a plurality of 
different modes of operation of a video guidance sensor system. After the overall 
system software performs basic functional testing upon powerup, the software- 
based method of the invention checks, for correctness, allowable commands for six 
different modes of operation. If a particular command checks out, the commanded 
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mode proceeds. As discussed below, an "acquire" or "acquisition" mode, wherein a 
target with an acceptable range is sought, and a "track" or "tracking" mode, wherein 
the acquired target is tracked, are the primary modes of operation for the video 
guidance sensor system. 

[0012] In accordance with the invention, there is provided a method for 
controlling operations in a video guidance sensor system wherein images of laser 
output signals transmitted by the system and returned from a target are captured 
and processed by the system to produce data used in tracking of the target, the 
target including a predetermined pattern of light reflectors so that a captured image 
of the target includes a pattern of light spots corresponding to the pattern of light 
reflectors, the method providing for performance of each of the following modes of 
operation in response to a corresponding mode command: 
[0013] (i) a reset mode wherein program integrity checks are performed 
and an output produced; 

[0014] (ii) a diagnostic mode wherein diagnostic operations are carried out 
and a known output is obtained from a previously synthesized image; 
[0015] (iii) a standby mode wherein status data is output at a fixed rate; 
[0016] (iv) an acquisition mode wherein range information is included in a 
corresponding acquisition mode command and an attempt is made, based on 
captured images of returned laser signals, to find a target pattern within a range 
window centered around an estimated range; 

[0017] (v) a tracking mode wherein images of a target pattern are captured 
and data relevant to that target is output; and 

[0018] (vi) a spot mode wherein captured images of returned laser signals 
are processed to produce data for all spots found in the image, 
[0019] and the method providing for automatic transition to the standby mode 
from the reset mode after the integrity checks are performed and from the diagnostic 
mode to the reset mode after the diagnostic operations are carried out, and 
[0020] the method permitting acceptance of acquisition, spot, reset and 
diagnostic commands only when the system is in the standby mode, and 
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[0021] the method further providing for automatic transition from the 

acquisition mode to the tracking mode when ah acceptable target pattern is found. 

[0022] In one preferred implementation, two different rates of tracking are 

provided in the tracking mode, and the tracking rate to be used is determined as part 

of an acquisition mode command. In an alternative preferred implementation, the 

tracking rate to be used is determined based on a separate tracking command. 
♦ 

[0023] Preferably, entry into the reset mode is limited to one of (i) powering up 
the video guidance sensor system and (ii) the system undergoing a soft reboot. 
[0024] Advantageously, in each of the modes, an output is produced which 
includes analog data corresponding to temperature and voltage levels of system 
hardware. 

[0025] In one advantageous implementation, the method is carried out using 
two different digital signal processors. Preferably, one of the processors (the 
command processor) receives said mode commands from an external source, 
processes the commands and passes the commands to the other of the processors 
(the image processor) and the other of said processors carries out image processing 
required by some of the commands. 

[0026] Further features and advantages of the present invention will be set forth 
in, or apparent from, the detailed description of preferred embodiments thereof which 
follows. 

Brief Description of the Drawings 

[0027] Figure 1 is state transition diagram illustrating transition relationships 
between the modes of operation of the video guidance sensor system; 
[0028] Figure 2 is the main flow chart for the command processor of the 
system; and 

[0029] Figure 3 is the main flowchart for the image processor of the system. 
Detailed Description of the Preferred Embodiments 

[0030] In general, the overall video guidance sensor software is designed to 
perform the communications and control for the sensor electronics hardware. In a 
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specific implementation, the hardware includes two digital signal processors (DSPs), 
a Texas Instruments F240 and a TMS320VC33 (CSS). The hardware also includes 
RAM and ROM for the two processors, two field-programmable gate arrays 
(FPGAs), image storage memory and camera and laser control electronics. Broadly 
speaking, the video guidance sensor software receives commands from an external 
source through a serial interface, checks the commands for correctness, implements 
the commands and returns the data appropriate for the command received. In the 
specific embodiment under consideration, the F240 processor receives and 
processes the commands and then passes the appropriate data to the CSS whose 
main function is to carry out the image processing necessary for some of the 
commands. 

[0031] As indicated above, an important function of the software of the 
invention is to process commands for and to control transitioning between six 
different modes of operation as follows: (i) reset; (ii) diagnostic; (ill) standby; (iv) 
acquisition; (v) track; and (vi) reset. Of these, diagnostic and reset are one-time 
events, i.e., when the software is commanded to do one of these modes, the 
software perfonns the indicated action and then automatically transitions into 
another mode. In each of the six modes, a part of the data output in that mode 
includes sixteen bytes of analog data corresponding to various temperatures and 
voltage levels in the system hardware. 

[0032] Briefly considering the various modes, in the reset mode, the software 
performs some basic integrity checks and outputs the integrity checks as well as the 
software version number. As indicated above, this is a one-time event in that after 
doing the above, the software automatically transitions into the standby mode. The 
reset mode is reached only after the entire video guidance sensor system is 
powered up or undergoes a soft reboot. When the system is commanded to reset, 
the system performs a soft reboot and comes up in the reset mode. 
[0033] Turning to the diagnostics mode, when commanded to do diagnostics, 
the software runs two diagnostic routines which involve firing the lasers, measuring 
output power and firing time, processing synthesized images and producing a known 
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output from the synthesized images. This process was generally described above. 
After the routines are run, the software transitions automatically into the reset mode, 
by rebooting the entire system. 

[0034] In the standby mode, the software outputs data as to its status at a 
predetermined rate which is 5 Hz in an exemplary embodiment. The reset and 
diagnostic commands are only accepted when the system is in the standby mode. 
[0035] When the system is commanded into the acquisition mode, a range 
estimate is included in the command. The overall software then provides for firing of 
the lasers, grabbing of the return images and processing of the images. Based on 
the processed images the software attempts to find a target pattern within a 
predetermined percentage of the estimated range (± 30% in an exemplary 
implementation). This process also occurs at a predetermined rate (e.g., 5 HZ, as 
above) and data is output that describes those target spots, if any, that are picked up 
by the sensor. If an acceptable target pattern is found, the software automatically 
transitions into the track mode. 

[0036] In the track or tracking mode, the sensor software again provides for 
firing the lasers, grabbing the returned images, processing the images and 
outputting the data relevant to the particular target. This data typically comprises 
range, bearing, attitude, and information about the spots. In the implementation 
being considered here, the sensor has two rates of tracking, viz., 5 Hz and 25 Hz. 
The tracking rate is given as part of the acquisition command or in a separate track 
command. The latter is only valid if the video guidance sensor system is already in 
the track mode. 

[0037] In the spot mode, the software provides for firing the lasers, grabbing 
images, processing the images and, in this case, outputting the data for the spots 
found in the image. This is again done at a fixed (5 Hz) rate. This mode is 
especially useful because many parameters of target acquisition can be varied, 
something that is necessary for calibration purposes on the ground. 
[0038] Referring to Figure 1 , a state transition diagram is shown for the six 
modes of operation. The reset, diagnostics, standby, spot, acquisition and tracking 
modes are indicated at A, B, C, D, E and F, respectively. A further "powerup" or 
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"watchdog tinier" mode of operation Is indicated at G. A watchdog timer timeout will 
transition from Initial "power up" to any of the other modes. 
[0039] The commands or other signals are as follows: automatic, denoted 1 ; 
reset command, denoted 2; standby command, denoted 3; acquire command, 
denoted 4; target not found, denoted 5; target found, denoted 6; spot command, 
denoted 7; and diagnostic command, denoted 8. 

[0040] As discussed above and is indicated in Figure 1 , the automatic 
transition 1 is between powerup G and reset mode A, reset mode A and standby 
mode C and diagnostic mode B and reset mode A. Thus, for example, as discussed 
hereinbefore, the reset mode A automatically transitions into the standby mode C, 
and the diagnostic mode B, after all routines are run, automatically transitions into 
the reset mode A. 

[0041] The reset command 2 represents the transition between the standby 
mode C and the reset mode A, while the standby command 3 represents the 
transition to the standby mode B from the spot mode D, the tracking mode F or the 
acquisition mode E, indicating a return to the standby mode C when the mode 
operations or routines for these modes are completed. 
[0042] The "acquire target" command 4 represents a transition to the 
acquisition mode E from the standby mode C, the spot mode D or the tracking mode 
F. 

[0043] The "target not found" signal represents the transition from the tracking 
mode F to acquisition mode E when no target is found during the tracking mode 
while, similarly, the "target found" signal 6 represents the transition between the 
acquisition mode G and the tracking mode F when a target is found. 

[0044] The spot command 7 represents the transition from the standby mode 
C or the acquisition mode E to the spot mode D. 

[0045] The diagnostic command 8 represents the transition from the standby 
mode C to the diagnostic mode B. 

[0046] Referring to Figure 2, there is shown the main flow chart for the above- 
mentioned F240 processor. As shown, the process proceeds from start 10, to an 
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"initialize everything" block 12 wherein the system is initialized to a "mode?" decision 
diamond 14 which determines the proper mode. Blocks 16 to 21 respectively 
represent the operations and routines for the reset, standby, diagnostic, spot, 
acquisition and track modes described above. 

[0047] Referring to Figure 3, there is shown the main flowchart for the image 
(C33) processor. As indicated above, the chief function of this processor is to carry 
out the image processing necessary for some of the commands. As illustrated, from 
start 24, the process proceeds to a "any command packet?" decision diamond 26 
and, if there is a command, to "mode?" decision diamond 28. As shown, three 
modes are posjsible, viz., standby, tracking and spot or acquire. 
[0048] For the standby mode, the program checksum is calculated and the 
standby packet is send, as indicated by block 30. 

[0049] In the spot (spot or acquire) mode, spots for the entire image are 
processed ("handled"), as indicated by block 32, and in a manner generally as 
described above in the introductory portion of this patent application. 
[0050] In the tracking mode, a "range?" decision diamond 34 has its output 
routed to one of a long-range target tracking operation, indicated by block 36, a 
short-range target racking operation, indicated by block 38 and "in-between" 
operation, which routes the output to an "overmode?" decision diamond 40. The 
latter provides three possibilities, viz., tracking of a long-range while finding a short- 
range target, as indicated by block 42, tracking a short-range target while finding a 
long-range target, indicated by block 44, and tracking both targets, indicated by 
block 46. 

[0051] Although the invention has been described above in relation to 
preferred embodiments thereof, it will be understood by those skilled in the art that 
variations and modifications can be effected in these preferred embodiments without 
departing from the scope and spirit of the invention. 
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